Most of the existing portrait segmentation algorithms ignore the hardware limitation of mobile devices and blindly pursue the effect, so that they cannot meet the segmentation speed requirement of mobile terminals. Therefore, a portrait segmentation network which could run efficiently on mobile devices was proposed. Firstly, the network was constructed based on the lightweight U-shaped architecture of encoder-decoder. Secondly, in order to make up for the fact that the Fully Convolutional Network (FCN) was limited by a small sensing domain, so that it was not able to fully capture the long-distance information, an Expectation Maximization Attention Unit (EMAU) was introduced after the encoder and before the decoder. Thirdly, for improving the accuracy of portrait boundary contour, a multi-layer boundary auxiliary loss was added at the training stage. Finally, the model was quantized and compressed. The proposed network was compared with other networks such as PortraitFCN+, ENet and BiSeNet on Veer dataset. Experimental results show that, the proposed network can improve the image reasoning speed and segmentation effect, as well as process the RGB images with the resolution of 224×224 at the accuracy of 95.57%.